Summary

Removes the ujson dependency and replaces it with the Python standard library json module across serialization, recorder, and zstack code paths.

Per OSV / GitHub Advisory DB, the previously pinned ujson==1.33 is affected by four known vulnerabilities, all resolved by the removal:

• CVE-2022-31116 (GHSA-wpqr-jcpx-745r) — incorrect handling of invalid surrogate pair characters (< 5.4.0)
• CVE-2022-31117 (GHSA-fm67-cv37-96ff) — potential double free of buffer during string decoding (< 5.4.0)
• CVE-2026-44660 (GHSA-c38f-wx89-p2xg) — memory leak in ujson.dump() on write failure (<= 5.12.0)
• CVE-2026-54911 (GHSA-3j69-69wj-xqx2) — malformed/truncated UTF-8 accepted and silently rewritten in ujson.dumps() (<= 5.12.1)

(CVE-2021-45958, cited in #1676, does not apply to the pinned version: its vulnerable range is >= 1.34, < 5.2.0. Removing the dependency moots it regardless.)

This continues the stalled work from #1676 by @PatStLouis:

• Rebased onto current main (the previously requested rebase).
• Fixes a logic bug that the original PR accidentally introduced (channel.py).
• Fixes two behavioral regressions of the ujson→stdlib swap itself (wire-format compactness, recorder bytes handling) and adds regression tests.

Ref: https://osv.dev/list?ecosystem=PyPI&q=ujson / https://security.snyk.io/package/pip/ujson

Changes

• Drop ujson from setup.py, build scripts, and dev-setup package lists.
• common/serializers/json_serializer.py: remove the try: import ujson branch; promote the existing OrderedJsonEncoder (stdlib json with sort_keys=True, separators=(',', ':'), ensure_ascii=False) to be the sole encoder, preserving sorted-key / compact output.
• Replace try: import ujson as json / except ImportError: import json fallbacks in recorder.py, test_recorder.py, stp_zmq/zstack.py, and scripts/test_zmq/.../zstack.py with a plain import json.
• stp_zmq/zstack.py (and the scripts/test_zmq copy): serializeMsg now passes separators=(',', ':'). ujson emitted compact JSON; stdlib's default separators insert whitespace, which grows every node-to-node wire message and breaks the exact-size assertions in stp_zmq/test/test_zstack.py:242/284/329 (calibrated to MSG_LEN_LIMIT with the 8-byte {'k':''} overhead of the compact form).
• plenum/recorder/recorder.py: add_to_store now passes default=json_default_bytes_to_str (a shared helper added to plenum/common/util.py) so bytes/bytearray are decoded to UTF-8 strings. SimpleZStackWithRecorder records raw wire frames — both the message and the ZMQ identity arrive as undecoded bytes — which ujson silently encoded as UTF-8 strings but stdlib json.dumps rejects with TypeError, so recording mode (STACK_COMPANION=1) crashed message servicing on the first frame. A bytes regression test is added (existing recorder tests only fed str).

Bug fix beyond #1676

In plenum/common/channel.py, the linting errors commit of #1676 changed:

-        if type(msg) != tuple:
+        if not isinstance(type(msg), tuple):

The intent was to silence flake8 E721 (do not compare types), which the original type(msg) != tuple triggers (E721 is not in this repo's .flake8 ignore list). However the replacement is always True: type(msg) is a class object, and a class object is never an instance of tuple. As a result, messages that were already tuples got wrapped again into a nested tuple — breaking handler routing in _process_sync — and it directly contradicts the adjacent comment explaining why isinstance must not be used here (it also matches NamedTuple).

This PR keeps the lint fix but does it correctly:

        if type(msg) is not tuple:

E721's own message recommends is / is not for exact type checks, so this form silences E721 and preserves the original semantics — it is not a revert of the linting fix. Verified with flake8 (--select=E721 reports no violation) and against tuple, NamedTuple, dict, str, and int inputs, where the result matches the original != behavior in every case:

Serialization compatibility

The previously pinned ujson==1.33 does not support the sort_keys kwarg — its C argument list is only obj, ensure_ascii, double_precision, encode_html_chars (verified against the 1.33 sdist from PyPI). The guard probe in json_serializer.py (uencode({...}, sort_keys=True)) therefore always raised TypeError and fell through to the stdlib OrderedJsonEncoder fallback. In other words: the signing/hashing serializer was already running on stdlib json in every standard install, and its byte output is unchanged by this PR. The new common/test/test_json_serializer_ujson_compat.py pins that byte format going forward (key ordering, compact separators, raw-UTF-8 non-ASCII, float repr, int-key coercion, top-level-bytes base64, and the nested-bytes TypeError characterisation).

The code paths where ujson was active are the ones without the probe (plain try: import ujson as json): the recorder and zstack. The two behavioral differences there — compact vs whitespace-padded output, and silent bytes-to-string coercion — are exactly what the separators=(',', ':') and recorder default= fixes above restore.

Notes

• pysha3 remains out of scope (per discussion in [Security] remove ujson package (CVE-2022-31116, CVE-2022-31117, CVE-2021-45958) #1676).

🤖 Generated with Claude Code